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(54) Vehicle front-view monitoring system 



(57) A vehicle front-view monitoring system takes 
fail-sate measures when a fail-safe measure-interrup- 
tion requirement using first parameter are met on a mon- 
itored image for a predetermined first period. The func- 
tions interrupted by the fail-safe measure is resumed 
when a fail-safe measure-release requirement using 



second parameter which is different from the first pa- 
rameter is met within a predetermined second period af- 
ter the fail-safe measure-interruption requirement has 
been met. The first period is variable in accordance with 
how accurately a lane marking on a road in the moni- 
tored image is recognized. 
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Description 

BACKGROUND OF THE INVENTION 

5 [0001] The present invention relates to a vehicle front-view monitoring system having a function of taking fail-safe 
measures against images irregularly monitored due to sunlight interference 

[0002] The vehicle front-view monitoring system having CCD cameras has recently attracted much attention. The 
system recognizes driving conditions sucn as a distance to a vehicle travelling ahead based on images taken by the 
cameras and warns a driver of danger or controls the vehicle such as by shiftdowns. 
io [0003] Recognition of the driving conditions using a stereo-distance measuring technique obtains a positional differ- 
ence (parallax) on an object in a pair of the images and calculates the distance to the vehicle travelling ahead by 
trigonometrical survey. 

[0004] Such a vehicle front-view monitoring system requires fail-safe measures for securing safe driving. A failure 
to be detected is a fault monitoring condition such that the vehicle in front is beyond recognition while monitoring 
*5 cameras are facing weak sunlight. 

[0005] Image monitoring failures are discussed with reference to FIGS! 1 to 3. 

[0006] When monitoring vehicles traveling ahead (FIG. 1 ) while facing weak sunlight, disc-like diffusion could appear 
on the monitored image, as illustrated on the left-upper side in FIG. 2, due to saturation of luminance. Such a disc-like 
diffusion appearing on the monitored image while the vehicle is facing the sunlight is called a sunlight-diffusion disc 

20 hereinafter. These sunlight-diffusion discs cause the saturation of pixel luminance on the disc areas and the surrounding 
areas. A correct image data therefore can not be obtained, or vehicles in front are masked by the sunlight-diffusion 
discs, which results in monitoring failures. Moreover, the disc edges have a big luminance change between adjacent 
pixels in the horizontal direction and hence. incorrect distance data could be calculated on the edges. 
[0007] FIG. 3 illustrates distance-data calculation based on stereo images (a reference image and an image to be 

25 compared such as shown in FIG. 1). 

[0008] In FIG. 3, black sections have a big luminance change between the adjacent pixels in the horizontal direction. 
These sections carry distance data (depth). The intensities of most pixels in the sunlight-diffusion disc area have been 
saturated, so that no distance data is calculated in this area (a lost state). Moreover, the disc edges have a big luminance 
change in the horizontal direction, so that the incorrect distance data could be calculated. 

30 [0009] Ensuring safe driving in such a fault condition due to a generation of the sunlight-diffusion discs discussed 
above requires the fail-safe measures to temporarily halt the monitoring function. 

SUMMARY OF THE INVENTION 

35 [001 0] A purpose of the present invention is to provide a vehicle front-view monitoring system for monitoring optical 
irregularities occurring on monitored wages and swiftly taking fail-safe measures. 

[001 1] Another purpose of the present invention is to provide a vehicle front-view monitoring system taking appro- 
priate fail-safe measures while avoiding repetition of unnecessary fail-safe measures. 

[0012] The present invention provides the vehicle front-view monitoring system for taking the fail-safe measures 
*o when a fail-safe measure-interruption requirement using a first parameter is met on a monitored image for a predeter- 
mined first period and resuming a function interrupted by the fail-safe measures when a fail-safe measure-release 
requirement using a second parameter different from the first parameter is met within a predetermined second period 
after the fail-safe measure-interruption requirement has been met. 

[0013] Moreover, the present invention provides a vehicle front-view monitoring system for taking a fail-safe measure 
45 when a fail-safe measure-interruption requirement is met on a monitored image for a predetermined first period and 
resuming a function interrupted by the fail-safe measures when a fail-safe measure-release requirement is met within 
a predetermined second period after the fail-safe measure-interruption requirement has been met, the first period being 
variable in accordance with an accuracy of a lane marking on a road in the monitored image is recognized. 
[001 4] By way of example only, a specific embodiment of the present invention will now be described, with reference 
50 to the accompanying drawings, in which :- 

FIG. 1 illustrates a normal image monitored on a monitor screen; 

FIG. 2 illustrates an irregular image monitored on the monitor screen on which sunlight-diffusion discs are gener- 
ated; 

55 FIG. 3 illustrates the irregular image monitored on the monitor screen, including inaccurate distance data due to 

generation of sunlight-diffusion discs; 

FIG. 4 is a block diagram of a preferred embodiment of the vehicle front-view monitoring system according to the 
present invention; 
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FIG. 5 shows a flow chart of a weak-sunlight-facing driving determination routine; 

FIG. 6 shows a flow chart of a weak-sunlight-facing driving determination routine; 

FIG. 7 illustrates the upper and lower areas set on the monitor screen; 

FIG. 8 illustrates an area on which a vehicle travelling ahead is monitored; 
5 FIG. 9 illustrates a monitoring area; 

FIG. 10 illustrates measured luminance-distribution characteristic parameters; 

FIG. 11 illustrates normalized luminance-distribution characteristic parameters; 

FIGS. 12A and 12B illustrate luminance addition characteristics, in normal driving; 

FIGS. 13A and 13B illustrate luminance addition characteristics in weak-sunlight-facing driving; 
10 PIGS. 14A to 14C illustrate luminance center; 

FIG. 15 shows a flow chart of a fail determination routine; 

FIG. 1 6 shows a flow chart of a count-up amount setting routine; 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

15 ~ ~ ~~ " . ~ '~ " 

[0015] Preferred embodiments according to the present invention will be disclosed with reference to the attached 
drawings. 

[0016] FIG. 4 shows a block diagram of a preferred embodiment of a vehicle front-view monitoring system according 
to the present invention. 

20 [001 7] A pair of cameras 1 and 2 having image sensors such as CCDs are attached to a vehicle such as an automobile 
in the vicinity of tyres with a predetermined distance (corresponding to camera-baseline length) in the direction of 
vehicle width, for taking images ahead. 

[0018] The main camera 1 (right camera) takes a reference image required for stereo processing whereas the sub- 
camera 2 (left camera) takes an image to be compared. 

25 [0019] Analog images taken by the cameras 1 and 2 while the main camera 1 and sub-camera 2 are synchronized 
with each other are converted into a specific luminance gradation (for example, gray scale at 256 gradations). 
[0020] The digitized images are processed by an image correcting section 5 for correction of luminance, image 
geometrical conversion, etc. An inevitable misalignment on the cameras 1 and 2 will affect left and right images. Such 
effect can be cancelled by geometrical conversion such as an image rotation and a parallel shift with Affine conversion, 

30 etc. Luminance data on the stereo images (a reference image and an image to be compared) for one frame corrected 
by the image correcting section 5 are stored in an image memory 8. 

[0021] The luminance data are further supplied to a stereo-image processor 6 for calculating distance data (parallax 
for each object). Each distance data corresponds to a positional deviation on the same object in the stereo images. 
An actual distance to the object is defined on the basis of parallax by a trigonometrical survey. The distance data for 
35 one frame thus obtained are stored in a distance-data memory 7. 

[0022] The luminance data and the distance data are supplied to a microcomputer 9. 

[0023] The microcomputer 9 includes a road-recognizing section 10, a three-dimensional (3-D) object-recognizing 
section 11 , a fail-determining section 12 and a processor 13. 

[0024] In response to the luminance and distance data, the road-recognizing section 1 0 recognizes the condition of 
*o a road ahead, and the 3-D object-recognizing section 11 recognizes three-dimensional objects (vehicles travelling 
ahead). 

[0025] Based on the data from the sections 1 0 and 11 , the processor 1 3 warns the driver of a danger via a warning 
device 1 9 such as a display or a speaker when it determines that the warning is necessary. 

[0026] Moreover, the processor 13 controls several controllers 14 to 18 if necessary. For instance, it sends a com- 
45 mand signal to an automatic transmission (AT) controller 14 for downshifts or to an engine controller 1 8 for an engine 
power-down. The processor 13 may further send the command signals to an antilock-braking system (ABS) controller 
15, a traction control system (TCS) controller 16 and a vehicle controller 17 for controlling an engine speed and a 
torque distribution to tyres, etc. 

[0027] The fail-determining section 12 determines whether the cameras 1 and 2 are taking the images into the sun 
so and a fail is occurring, based on the luminance and the distance data stored in the image memory 8. The vehicle and 
the warning controls are interrupted by the fail-safe measures for prevention of malfunction which could otherwise 
occur due to lowering in monitoring accuracy during a period of determination in that a fail is occurring, or of "1" set 
for a fail flag NG which will be described later. 

[0028] FIGS. 5 and 6 show flowcharts for indicating a routine of determining whether the cameras 1 and 2 are taking 
55 images into the sun while the vehicle is facing the sunlight, which is called a sunlight-facing driving hereinafter. The 
routine is repeated by the fail-determining section 12 by a predetermined interval (a cycle of 100 ms for example). 
[0029] The requirements for taking the fail-safe measures (called fail-safe measure-interruption requirements here- 
inafter) are discussed before explaining the routine in detail. 
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[0030] Requirements for steps S1 to S6 are necessary for taking the fail-safe measures in the present invention. It 
is determined that the cameras 1 and 2 are taking the images into the sun only when all the fail-safe measure-inter- 
ruption requirements are met in steps S1 to S6 and the routine proceeds to step S1 2 whereas to step S7 if not met in 
any one of steps S1 to S6. 

[0031] As disclosed later, the fail determining section 12 determines whether the fail-safe measure-interruption re- 
quirement or a fail-safe measure-release requirement is met and performs counter increment or decrement in accord- 
ance with a result of determination. The fail-safe measure-release requirements are the requirements for resuming 
functions that have been interrupted by taking the fail-safe measures. 

[0032] The interruption requirements are required for determining whether the cameras 1 and 2 are taking images 
into the sun, so that the fail-safe measures should be taken. A counter increment is performed when an interruption 
requirement is met. The fail-safe measures will never be taken before the counter reaches a predetermined value, 
even if the interruption requirements are met. 

[0033] The release requirements are required for determining whether to halt the counter increment. In the present 
invention, the counter starts a count-up when an interruption requirement is met and continues until a release require- 
ment is met even though monitoring situation is changing and no interruption requirement is met anymore. 
[0034] The present invention deploys the following six interruption requirements for taking fail-safe measures, how- 
ever, any combination of' at least two of the requirements or addition of other requirements are available. 
[0035] First Requirement: No vehicle is recognized ahead, or no distance to a vehicle in front is calculated (step S1 ). 
[0036] The fail-safe measures will be taken in the present invention when the cameras 1 and 2 are taking the images 
into the sun (sunlight-facing driving condition) so that the vehicle that actually traveling ahead is lost on a monitor 
screen of the monitoring system. In other words, the fail-safe measures will not be taken even though the sunlight- 
diffusion discs shown in FIG. 2 are appearing while the vehicle traveling ahead is recognized; 
[0037] The present invention therefore deploys a decision in that no vehicles are recognized ahead as a fail-safe 
measure-interruption requirement. This decision step is executed even though no vehicles actually exist ahead. 
[0038] When the requirement in step S1 (FIG. 5) is met, the routine proceeds to step S2 to determine whether the 
cameras 1 and 2 are taking the images into the sun (sunlight-facing driving condition). 

[0039] Calculation of the distance to the vehicle in front in step S1 is explained before disclosing the step S2. 
[0040] The distance to the vehicle in front is calculated in combination of a stereo-distance measuring technique and 
an image-recognition technique. 

[0041] Calculated first is a lane model that indicates a three-dimensional shape of a road ahead. Lane markings 
painted on the left and right sides of a lane and displayed on the monitor screen have the luminance higher than that 
of the road surface. The markings are extracted as areas having luminance edges on the borders between that and 
the road surface. 

[0042] The parallax on the lane thus extracted is calculated for each (4 x 4)-pixel block, for example. A two-dimen- 
sional position and the parallax in the monitored image for one pixel block in which the lane is displayed provide a 
three-dimensional position and a distance of the lane portion by using known coordinate conversion formulas. 
[0043] The calculated three-dimensional lane distribution is approximated into a straight line for a predetermined 
segment. The straight lines are connected into a polygonal line for calculating the lane model. 
[0044] The vehicle travelling ahead is recognized as a three-dimensional object existing on the area (road surface) 
surrounded by left and right lane models. The distance to the vehicle in front is calculated based on the parallax of the 
object. 

[0045] These techniques are disclosed in detail in Japanese Patent Application No. 11-269578 (road recognition) 
and Japanese Unexamined Patent Publication No. 5-265547 (recognition of vehicle traveling ahead). 
[0046] Second Requirement: A shutter speed "a N1 is equal to or lowerthan a predetermined reference speed (step S2). 
[0047] The shutter speed tends to be higher in a sunlight-facing driving condition in which the vehicle in front is 
missing on the monitor screen than that in a usual driving condition. In view of this tendency, an appropriate reference 
shutter speed such as 2000 u. sec is set. 

[0048] The present invention deploys a decision in that the shutter speed "a" is lower than this reference shutter 
speed as a fail-safe measure-interruption requirement. 

[0049] The shutter speed, however, tends to be extremely higher against a strong sunlight in mid-summer, for ex- 
ample 200 ji sec. or lower. In view of this tendency, the routine proceeds to step S3 (FIG. 5) when the requirements 
in steps S1 and S2 are both met. 

[0050] Third Requirement: The upper luminance saturation factor LR1 is larger than the lower luminance saturation 
factor LR2 (step S3). 

[0051] Sunlight-diffusion discs images shown in FIG. 2 will be generated on the upper side of the monitor screen 
when a vehicle in front is missing during sunlight-facing driving. This phenomenon has a tendency in thatthe luminance 
saturates on the upper side. 

[0052] In view of this tendency, as illustrated in FIG. 7, the upper and the lower areas are provided in a reference 
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image. 

[0053] The present invention deploys the decision in that the upper luminance saturation factor LR1 in the upper 
area is larger than that of the lower luminance saturation factor LR2 in the lower area as a fail-safe measure-interruption 
requirement. 

5 [0054] The luminance saturation factors LR1 and LR2 correspond to a ratio of the number of pixels for which lumi- 
nance has saturated (for example, the luminance 240 or more) to the total number of pixels in the upper and the lower 
areas, respectively. 

[0055] The requirement in step S3 could be met even though no sunlight-diffusion discs (FIG. 2) are generated e.g. 
when there is no shielding in a fair day or while travelling on a bridge. In view of this, the routine proceeds to step S4 
10 (FIG, 5) when the requirements in steps S1 , S2 and S3 are all met. 

[0056] No reference values are set for the upper and lower luminance saturation factors LR1 and LR2 in this em- 
bodiment. This is because factors tend to vary very largely according to the driving conditions. Particularly, the lower 
factor LR2 often becomes zero in a normal driving condition. 

[0057] Fourth Reqqireaent: The number of luminance edges EN for an area of the vehicle in front is smaller than a 

15 predetermined reference number (step S4). 

[0058] While the vehicle travelling ahead is missing on the monitor screen during the sunlight-facing driving, the 
number of the luminance edges EN on the sunlight-diffusion disc area (FIG. 2) tends to become extremely small. 
[0059] The number of the luminance edges EN indicates the number of a pair of adjacent pixels in the horizontal 
direction on the monitor screen for which variation in luminance is more than a predetermined reference value. 

20 [0060] As illustrated in FIG. 8, a vehicle in front area is set in the upper-mid section of the monitor screen, in which 
the vehicle travelling ahead would be displayed. 

[0061] The present invention deploys the decision in that the number of the luminance edges EN in the vehicle in 
front area (FIG. 5) is smaller than an appropriate reference number as a fail-safe measure-interruption requirement. 
[0062] This requirement could, however, .be met in driving at night, in tunnel, mist, smoke or on a snowy road. In 

25 view of this, the routine proceeds to step S5 (FIG. 5) when the requirements in steps S1 , S2, S3 and S4 are all met. 
[0063] The number of the calculation of the distance data in the vehicle in front area can be used instead of the 
number of the luminance edges EN. This is because the number of the luminance edges EN and the number of the 
calculation of the distance data have a strong correlation in a stereo-matching, orthe larger the number of the luminance 
edges EN, the larger the number of the calculation of the distance data. 

30 [0064] Fifth Requirement: The luminance-distribution-characteristics parameter normalized by using a shutter speed 
"a" meets a requirement. 

[0065] As shown in FIG. 9, several slit-like monitoring areas Ni are arranged on the upper side of the monitor screen 
in the horizontal direction. The reason for the arrangement of monitoring areas Ni on the upper side is that the luminance 
mostly varies on the upper side when the vehicle is facing a weak sunlight. 
35 [0066] The monitoring areas Ni are provided for evaluating the luminance-distribution characteristics for a monitored 
image in the horizontal direction. 

[0067] For example, a reference image is divided into blocks of 16 pixels each in the horizontal direction when the 
reference image is composed of 200 and 512 pixels in the vertical and horizontal directions, respectively. Therefore, 
32 monitoring- areas Ni of (200 x 16) pixels are Set. 
40 [0068] Calculated next is addition of luminance Ai for each monitoring area Ni. The addition of luminance Ai for a 
monitoring area Ni can be calculated by adding the luminance of sample pixels that are uniformly distributed over the 
monitoring area Hi. The added luminance of the sample pixels may further be averaged. 

[0069] The addition of luminance Ai is the value, for example, obtained by adding the luminance of 4 pixels sampled 
in the horizontal and vertical directions each. Pixels uniformly distributed over the monitoring area Ni can be used for 
45 calculating the luminance over the area Ni with less amount of calculation. The addition of luminance Ai can be calcu- 
lated for all pixels involved in the area Ni if the amount of calculation is out of the question. 

[0070] The Ai-distribution characteristics for the 32 monitoring areas corresponds to the image distribution charac- 
teristics in the horizontal (lateral) direction. 

[0071] The parameter (luminance-distribution characteristic value) that indicates luminance-distribution character- 
50 istics on the addition of luminance Ai is normalized by a shutter speed "a". .Used here as the parameter is luminance- 
addition variance VAR that indicates a variance in the addition of luminance Ai. Luminance-addition variance VAR' 
normalized by the shutter speed "a", called a normalized luminance-addition variance VAR', is calculated by the fol- 
lowing equations: 

55 

VAR' = VAR/a 
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VAR 2 - { (A1 - A ave ) 2 + (A2- A ave ) 2 + ... (AN - A ave ) 2 }/n = I (Ai 2 )/n -(£Ai/n) 2 (1) 

[0072] The normalized luminance-addition variance VAR' as the luminance-distribution-characteristics parameter 
requires a decision in that the VAR* is larger than an appropriate reference value as of fail-safe measure-interruption 
requirement. 

[0073] Instead of the VAR' , the maximum addition of luminance Amax that indicates the maximum (peak) of the 
addition of luminance Ai can be used as the luminance-distribution-characteristics parameter. 
[0074] The Amax requires a decision in that the maximum addition of luminance Amax ' normalized by the shutter 
speed M a", called normalized maximum addition of luminance Amax ', is larger than an appropriate reference value as 
a fail-safe measure-interruption requirement. - 

[0075] Another decision in that both the normalized VAR' and the normalized Amax' are larger than the corresponding 
appropriate reference values can be used as a fail-safe measure-interruption requirement. 

[0076] The reason for using the normalized VAR' and Amax' for determination of whether the vehicle is facing the 
weak sunlight is discussed with reference to FIGS. 10 and 11. 

[0077] FIG. 10 shows the un-normalized characteristics of the luminance-distribution-characteristics parameters 
(VAR and Amax) that were measured in an actual driving test. Images were taken into the weak sunlight from about 
300 sec. to about 340 sec. after the start of measurement. 

[0078] The measurement of Amax teaches that Amax was measured in a fixed range through the test and hence it 
is difficult to find out an extreme tendency on a weak-sunlight-facing driving. 

[0079] The measurement of VAR shows a peak on about 270 sec. after the start of measurement, which appeared 
while the vehicle was not facing the sun, and hence it is also difficult to find out an extreme tendency on the weak- 
sunlight-facing driving. 

[0080] FIG. 1 1 shows the normalized characteristics of the luminance-distribution characteristics parameters (VAR' 
and Amax '). 

[0081] compared to the test results shown in FIG. 10, the results in FIG. 11 show an extreme tendency in which both 
VAR' and Amax ' are extremely increased while the vehicle is facing the weak sunlight. 

[0082] It is therefore apparent that these parameters VAR' and Amax' can be used effectively for determining whether 
the vehicle is lacing the weak sunlight. 

[0083] Accordingly, the present invention employs the parameters Amax ' and VAR' that are obtained by normalizing 
the parameters Amax and VAR with a shutter speed "a" for determination of the weak sunlight-facing driving in view 
of the fact that Amax and VAR are the functions of the shutter speed "a". 

[0084] The decision in that both Amax ' and VAR' are larger than the corresponding reference values as a fail-safe 
measure-interruption requirement further enhances accuracy of determination on fail-safe measurements. 
[0085] The shutter speed, however, tends to be higher in a fair day, and hence the routine proceeds to step S6 (FIG. 
5) when the requirements in steps S1 , S2, S3, S4 and S5 are all met. 

[0086] Sixth requirement: Division of the luminance moment by the total addition of the luminance is equal to or lower 
than a reference value (step S6). 

[0087] Step S6 evaluates the distribution of addition of luminance Ai for each monitoring area Ni from the point of 
view which is different from step S5. 

[0088] FIGS. 12A and 12B show the distribution characteristics of the addition of luminance Ai in regular conditions. 
[0089] In detail, FIG. 12A shows an example of the distribution characteristics for a driving condition such that a 
vehicle is entering a freeway, so that the sky is displayed all over the upper section of the monitor screen. 
[0090] FIG. 12B shows another example of the distribution characteristics for a driving condition such that the vehicle 
is travelling on a road uncovered with snow, which is displayed in the mid section of the monitoring screen, while the 
ground besides the road has been covered with snow. 

[0091] On the other hand, FIGS. 13A and 13B show the distribution characteristics of the addition of luminance Ai 
while the cameras 1 and 2 are facing a weak sunlight during the sunlight-facing driving. 

[0092] The weak-sunlight-facing driving results in a tendency in that high luminance portions being converged on 
one section corresponding to the sunlight-diffusion disc (FIG. 2) of a monitored image in the horizontal direction. This 
is shown in FIG. 13A in which the high luminance portions are converged on the mid section of the monitor screen 
FIG. 13B shows high luminance portions converged on the right section of the monitor screen. 
[0093] Step S6 determines whether the present driving condition is a regular one such as shown in FIG. 1.2A or 1 2B 
or the weak-sunlight-facing driving such as shown in FIG. 13A or 13B. 

[0094] For this determination, a luminance center LC and a luminance moment LM are introduced as defined in the 
following equations: 
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LC = E(i x AiyiAi 



UvUZ(li-LCIxAi) 



(2) 



where T indicates a monitoring area Ni- number {0 to 31 ) and Ai indicates an addition of the luminance Ai in the 
i-th monitoring area Ni, like in step S5. 

[0095] The luminance center LC is defined as a horizontal position at which the addition of luminance Ai in the 
io horizontal direction are converged. The LC is obtained by dividing the total of the product of Ai x " t" ^horizontal position) 
by the total of Ai. 

[0096] For example, when all of the addition of luminance Ai are the same value as shown in FIG. 14A, the luminance 
center LC is 15. 5 that lies between 0 and 31 . Moreover, when the addition of luminance AO to A15 are all zero and 
A16 to A31 are the same value (* 0), the luminance center LC is 23. 5. 
15 [0097] The luminance moment LM is defined as the total of the product of a distance from the luminance center LC 
to a horizontal position "i" times the addition of luminance Ai. The moment LM indicates the distribution of the addition 
. of luminance Ai in the horizontal direction on the basis of the luminance center LC. The luminance moment LM becomes 
small when the addition of luminance Ai is small in overall of the luminance is converged in the vicinity of the luminance 
center LC. 

20 [0098] Calculated next is the value obtained by dividing the luminance moment LC by the total of addition of luminance 
XAi, or luminance moment LM' that is normalized by 1A\ as indicated by the following equation: 

LM' w LM/ZAi 

25 « I ( | i - LC | x Ai) ZAi. ... (3) 

[0099] In normal driving at the distripution characteristics such as shown in FIG. 14A, the luminance moment LM 
tends to become large because luminance are distributed all over in the horizontal direction. 
30 [0100] on the other hand, in the sunlight-facing driving at the distribution characteristics such as shown in FIG. 1 4C, 
the luminance moment LM tends to become small because the luminances are converged in the vicinity of the luminance 
center LC. 

[0101] However, it is sometimes difficult to discriminate the sunlight-facing driving condition from a normal driving 
condition because the luminance moment LM becomes large for a bright image even in sunlight-facing driving. 
35 [0102] In view of this, the normalized luminance moment LM* that is obtained by dividing the luminance moment LM 
by the total of addition of luminance Ai and that depends only on luminance distribution is used for the determination 
in step S6. 

[0103] The normalized luminance moment LM' has a tendency of becoming large for the distribution characteristics 
shown in FIGS. 12A and 12B whereas small in FIGS. 13A and 13B. 
40 [0104] In view of this, step S6 compares the normalized luminance moment the LM 1 with an appropriate reference 
value for detecting the weak-sunlight-facing driving condition that- is shown in FIGS. 13A and 13B. 
[0105] The equation (3) can be modified for a large weight of the addition of luminance Ai for evaluation of Ai distri- 
bution as follows: 

45 • 2 

LM' = L(li - LC I x Ai ) / LAi 

LC =E (i x - Ai 2 )/ ZAi (4) 

50 

[0106] In FIG. 5, when all the requirements in steps S1 to S6 are met, or all the fail-safe measure-interruption re- 
quirements are met, it is determined that the vehicle is facing a weak sunlight. A count flag Fcnt is then set at "1" (step 
S12). The count flag Fcnt is set at "0" when a counter K reaches the lower limit value o whereas "1 " while the counter 
K is counting up or down (K * 0). 
55 [0107] The counter K is set at a count value a (> 0) in the following step S13. 

[0108] The counter K is used for a fail determining routine such as shown in FIG. 1 5 for determining whether to take 
fail-safe measures. 
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[0109] When the counter K reaches the upper limit value Klim (step S30) due to increment in the weak-sunlight- 
facing driving condition determining routine, a fail flag NG is set at "1" (step S31). On the other hand, when the counter 
K reaches the lower limit value 0 (step S32), the count flag Fcnt and the fail flag NG are both reset at "0" (steps S33 
andS34). ■ 
5 [0110] Moreover, when the counter K is in the range from the lower limit value 0 to the upper limit value Klim. the 
routine ends via steps S30 and S32 with no change in the count flag Fcnt and the fail flag NG. 
[0111] while a front-view monitoring is interrupted by the fail-safe measures (NG = 1), it will not be resumed (fail- 
safe measures will not be released) unless the counter K reaches the lower limit value 0. 

[0112] on the other hand, while the front-view monitoring is performed (MG = 0), it will not be interrupted by the fail- 
io safe measures unless the counter K reaches the upper lower limit value Klim. 

* [01 1 3] The counter K starts the count-up when a fail-safe measure-interruption requirement is met. The fail flag NG 

is set at "1" when such a driving condition continues for a predetermined period (decided based on the upper limit 

value Klim and the count-up amount a ) for taking the fail-safe measures. Monitoring is thus interrupted for avoiding a 

decrease in monitoring control accuracy in the weak-sunlight-facing driving. 
15 [0114] The count-up amount a is decided according to a count-up amount setting routine shown in FIG. 6. The 

amount a is decided in accordance with a distance variable and a difference in luminance in left and right cameras 

(step S20). 

[0115] Usually cameras are facing the sunlight in the sunlight-facing driving and hence the variation in calculation 
of the distance tends to be larger compared to a normal driving condition. 
20 [0116] In view of such a tendency, the present invention obtains the distance variable in a predetermined area in a 
monitored image, for example, an area extending in the vertical direction in the image. The count-up amount a is set 
at a large one when the variable is large. It is also set at a large amount in the sunlight-facing driving due to a larger 
luminance difference between the left and right cameras compared to the normal driving. 

[011 7] On the other hand, it is determined that the vehicle (the cameras 1 and 2) is not facing the sunlight if any one 
25 (or more) of the fail-safe measure-interruption requirements is (or are) not met through steps S1 to S6 in FIG. 5. 

[0118] The routine then proceeds to step S7 in FIG. 6 to determine whether the count flag Fcnt has been set at "1", 
or the counter K has been counting up or down in Step S7. If it is determined as K = 0 : the routines will not proceed 
anymore and ends. 

[0119] As disclosed, monitoring is continued without taking any fail-safe measurements during the normal driving 
30 condition without facing sunlight and the count K having reached the lower limit value 0, or the fail flag NG having been 
set af'O". 

[0120] On the contrary, when any one; (or more) of the fail-safe measure-interruption requirements is (or are) not 
met through steps S1 to S6, but it is " YES" (Fcnt=1 , K * 0) in step S7, the routine proceeds to step S8. 
[0121] Steps S8 to S10 define fail-safe measure-release requirements for the counter K to count up or down to 
35 resume functions that have been interrupted by the fail-safe measures. The counter K will count down when a fail-safe 
measure-release requirement is met whereas count up when a requirement is not met. 

[0122] The fail-safe measure-release requirements are not as severe as the fail-safe measure-interruption require- 
ments for avoiding repetition of the fail-safe measures while it is repeated that the vehicle travelling ahead is lost and 
appeared on the monitoring screen. 
40 [0123] As one of the fail-safe measure-release requirements, it is determined in step SB whether the shutter speed 
"a" is 3000 n sec. or lower. If not, or this fail-safe measure-release requirement is met : the routine proceeds to step 
S11 in which the counter K counts down by a count-down amount p. 

[0124] On the other hand, if " YES" in step S8, the routine proceeds to step S9 to determine whether the luminance- 
addition variance VAR is equal to or larger than the reference value. 
45 [0125] The VTR tends to be large in the sunlight-facing driving condition because, particularly, when facing a strong 
sunlight, luminance-saturated sections appear on the monitor screen due to smears or sunlight-diffusion discs (FIG. 
2) and also blackout sections. 

[0126] However, VTR tends to be large not only in the sunlight-facing driving, but also, for instance, in a sunny day, 
due to a strong contrast between sunny and shade places. It tends to be large, particularly, there is the shade of guard 

50 rails or roadside trees along the road. 

[0127] In view of such a tendency, the present invention deploys the luminance-addition variance VTR that will not 
vary so much in the weak-sunlight-facing driving compared to the normalized luminance-addition variance VTR', for 
the fail-safe measure-release requirements less severe than the fail-safe measure-interruption requirements, to avoid 
the repetition of fail-safe measures discussed above. 

55 [0128] If " YES" in step S9 (FIG. 6), the routine proceeds to step S13 (FIG. 5) in which the counter K starts count- 
up. On the other hand, if "NO" in step S9, the routine proceeds to step S10 to determine whether lane markings have 
been accurately recognized. Accuracy on a lane marking recognition can be used as a fail-safe measure-release 
requirement for avoiding the repetition of the fail-safe measures discussed above. This is because the road surface 
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will be mostly correctly displayed on the monitor screen and hence the lane marking recognition accuracy will not be 
lowered so much even in the weak-sunlight-facing driving. 

[0129] If the lane marking recognition accuracy has been lowered in step S10, the routine proceeds to step S13 
(FIG. 5) in which the counter K starts count-up. 

[0130] On the other hand, if the lane marking recognition accuracy is not lowered in step S1 0, the routine proceeds 
to step S11 in which the counter K starts count-down. 

[0131] The lane marking recognition accuracy can be determined as disclosed in Japanese Patent Application NO. 
11-269578. The accuracy of lane markings detected on a monitored image is determined based on the number of 
edges of luminance on the lane markings, the positional relationship (continuity) to the lane markings detected previ- 
ously and parallelism between left and right lane markings. High lane marking recognition accuracy is achieved as the 
number of edges of luminance is larger the lane markings detected in the present image more continues to the pre- 
viously detected markings and the detected left and right markings are more parallel to each other. 
[01 32] As clearly understood from the foregoing disclosure, the fail-safe measure-release requirements (count-down 
requirements for the counter K) are as follows: 

(1 ) the shutter speed "a" is higher than 3000 u. se. , or 

(2) if not, the luminance-addition variance VAR is smaller than the reference value and lane marking recognition 
is accurate. 

[0133] As disclosed : the present invention monitors parameters such as the luminance-distribution characteristics 
parameter, the number of edges of luminance, the upper and lower luminance saturation factors and the luminance 
moment for detection of a weak-sunlight-facing driving condition. 

[0134] The fail-safe measures can therefore be taken exactly during the weak-sunlight-facing driving, thus offering 
a highly reliable vehicle front-view monitoring system. 

[0135] Particularly, in the present invention, several monitoring areas are set on the monitor screen, which are slit- 
like areas extending vertically on a monitored image for obtaining the luminance-distribution characteristic values (the 
luminance-addition variance VAR and the maximum addition of luminance Amax) in the horizontal direction of the 
image. 

[0136] The luminance-distribution characteristic values are normalized using a shutter speed "a" for obtaining pa- 
rameters (the luminance-addition variance VAR' and the normalized maximum addition of luminance Amax') that are 
highly related to weak-sunlight-facing driving. 

[0137] The normalized parameters VAR' and Amax' can be used for detection of a weak-sunlight-facing condition, 
which is more accurate than using the un-normalized parameters VAR and Amax as the fail-safe measure-interruption 
requirements. 

[0138] The fail-safe measure-release requirements are set as including several conditions in addition to a weak- 
sunlight-facing driving condition so that it is not so severe than those of the fail-safe measure-interruption requirements. 
[0139] This requirement setting avoids the repetition of unnecessary fail-safe measures such as the warning which 
could irritate the driver while it is repeated that the vehicle travelling ahead is lost and appeared on the monitoring 
screen. 

[0140] The fail-safe measure-interruption requirements include the normalized Luminance-addition characteristic 
values VAR' and Amax' whereas the fail-safe measure-release requirements include VAR and Amax , the parameters 
different from VAR' and Amax because those are not normalized. 

[0141] Moreover, the fail-safe measure-release requirements include the accuracy on the lane marking recognition 
that is not deployed for the fail-safe measure-interruption requirements. 

[0142] Deploying different parameters for the fail-safe measure-interruption/release requirements offers flexible fail- 
safe requirement settings that match an actual driving condition compared to setting in which reference values are 
changed between the interruption/release requirements for the same parameters. 

[0143] The count-up amount a is adjusted based on the luminance difference between the left and right cameras 
and distance data variable. 

[0144] This adjustment offers variable settings for a fail-safe determination time from a moment at which the fail-safe 
measure-interruption requirements are met to a moment of taking fail-safe measures in accordance with actual driving 
conditions, 

[0145] Disclosed so far is the vehicle front-view monitoring system that takes fail-safe measures against a weak- 
sunlight-driving condition in which the vehicle is travelling as facing the weak sunlight so that sunlight-diffusion discs 
appear on the monitor screen . 

[0146] However, the present invention is applicable to not only such a weak-sunlight-driving condition but also other 
optical phenomena such as smear occurring on a monitoring screen. 

[0147] As disclosed, according to the present invention, optical irregularities occurring on the monitored images due 
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to optical interfering can be accurately detected, thus, the fail-safe measures can be taken swiftly in response to such 
optical phenomena for securing the safety driving. 

[0148] Moreover, different parameters for the fail-safe measure-interruption/release requirements offer flexible fail- 
safe requirement settings that match actual driving conditions. 

[0149] It is further understood by those skilled in the art that the foregoing description is a preferred embodiment of 
the disclosed device and that various change and modification may be made in the invention without departing from 
the scope thereof. 



Claims 

1 . A vehicle front-view monitoring system for taking fail-safe measures when a fail-safe measure-interruption require- 
ment using a first parameter is met on a monitored image for a predetermined first period and resuming a function 
interrupted by the fail-safe measures when a fail-safe measure-release requirement using a second parameter 
different from the first parameter is met within a predetermined second period after the fail-safe measure-interrup- 
tion requirement has been met. 

2. The vehicle front-view monitoring system according to claim 1 , wherein the first parameter includes at least a value 
obtained by normalizing an addition of a luminance-characteristic value on the monitored image by a shutter speed 
for a camera device via which the image is monitored and the number of luminance edges on the monitored image 
and the second parameter includes at (east the shutter speed and the addition of a luminance- characteristic value. 

3. The vehicle front-view monitoring system according to claim 1 or claim 2, wherein the first and the second param- 
eters include parameters related to luminance-distribution characteristics on the monitored image but different 
from each other. 

4. The vehicle front-view monitoring system according to claim 3, wherein a parameter related to the luminance- 
distribution characteristics and involved in the first parameter is a value obtained by normalizing a luminance- 
addition variance or the maximum addition of a luminance on the monitored image by a shutter speed for a camera 
device via which the image is monitored whereas a parameter related to the luminance-distribution characteristics 
and involved in the second parameter is the luminance-addition variance. 

i 

5. A vehicle front-view monitoring system for taking a fail-safe measure when a fail-safe measure-interruption re- 
quirement is met on a monitored image for a predetermined first period and resuming a function interrupted by the 
fail-safe measures when a fail-safe measure-release requirement is met within a predetermined second period 
after the fail-safe measure-interruption requirement has been met, the first period being variable in accordance 
with an accuracy of a lane marking on a road in the monitored image is recognized. 
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